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Abstract: 

In today's marketplace, the cost per Terabyte of disks with EIDE interfaces is about 
a third that of disks with SCSI. Hence, three times as many particle physics events 
could be put online with EIDE. The modern EIDE interface includes many of the 
performance features that appeared earlier in SCSI. EIDE bus speeds approach 33 
Megabytes/s and need only be shared between two disks rather than seven disks. 
The internal I/O rate of very fast (and expensive) SCSI disks is only 50 per cent 
greater than EIDE disks. Hence, two EIDE disks whose combined cost is much less 
than one very fast SCSI disk can actually give more data throughput due to the 
advantage of multiple spindles and head actuators. We explore the use of 12 and 16 
Gigabyte EIDE disks with motherboard and PCI bus card interfaces on a number of 
operating systems and CPUs. These include Red Hat Linux and Windows 95/98 on 
a Pentium, MacOS and Apple's Rhapsody/NeXT/UNIX on a PowerPC, and Sun 
Solaris on a UltraSparc 10 workstation. 
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Introduction 



In today's marketplace, the cost per Terabyte of disks with EIDE (Enhanced In- 
tegrated Drive Electronics) interfaces is about a third that of disks with SCSI (Small 
Computer System Interface). Hence, three times as many particle physics events could 
be put online with EIDE. The modern EIDE interface includes many of the performance 
features that appeared earlier in SCSI. EIDE bus speeds approach 33 Megabytes/s and 
need only be shared between two EIDE disks rather than seven SCSI disks. The in- 
ternal I/O rate of very fast (and expensive) SCSI disks is only 50 percent greater than 
EIDE disks. Direct Memory Access (DMA), scatter/gather data transfers without in- 
tervention of the Central Processor Unit (CPU), elevator seeks, and command queuing 
are now available for EIDE, as well as support for disks larger than 8.4 Gigabytes. PCI 
(Peripheral Control Interface) cards allow the addition of even more EIDE interfaces, 
in addition to those already on the motherboard. 

Motivation 

There are a number of High Energy Physics Experiments that have produced Ter- 
abytes of data [1]. A few examples as of 12/95 are: 



Experiment 


Data set (Terabytes) 


FNAL-E791 


50 


FNAL-DO 


40 


FNAL-CDF 


10 


HERA-ZEUS 


5 


CESR-CLEO 


5 


LEP-Delphi 


5 


LEP-L3 


3.4 


HERA-HI 


2.5 


LEP-Aleph 


1.7 


LEP-OPAL 


1.5 



The efficiency of data analysis is greatly enhanced by using disk based files of 
filtered Data Summary Tapes (DSTs) rather than continually loading files from tapes. 
However, the high cost of disks have hindered more widespread use. Low cost EIDE 
disks are improving this situation. 
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Big Disks 



EIDE Disk Model 


Bigfoot 


Deskstar 


Deskstar 


Diamond 


Manufacturer 


Quantum 


IBM 


IBM 


Maxtor 


Capacity (Gigabytes) 


12 


16.8 


14.4 


17.2 


Max. Internal I/O (Mbits/s) 


142 


162 


174 




Avg. Seek Time (ms) 


12.0 


9.5 


9.5 


9.0 


RPM 


4000 


5400 


7200 


5400 


Unit Street Cost 


$241 


$407 


$407 


$410 


Cost $/Terabyte 


$20000 


$24000 


$28000 


$24000 



SCSI Disk Model 


Cheetah 


Ultrastar 


Manufacturer 


Seagate 


IBM 


Capacity (Gigabytes) 


18.2 


18 


Max. Internal I/O (Mbits/s) 


231 


180 


Avg. Seek Time (ms) 


5.7 


6.5 


RPM 


10000 


7200 


Unit Street Cost 


$1242 


$1000 


Cost $/Terabyte 


$68000 


$55000 



Tests Performed 

For this paper we tested two of the large capacity EIDE disks with six different 
operating systems and a PCI EIDE disk controller card. The six operating systems are 
Mac OS 8.1, Apple Rhapsody DR2, Sun Solaris 2.6, Windows 95b, Windows 98, and 
RedHat LINUX 5.1 (kernel 2.0.34). The two disk drives and the disk controller card 
are described below: 

• Quantum Bigfoot™ TX [2] 12 GB, 4000 RPM, 142 Mbits/sec Maximum 
internal data rate, 12 ms average seek time. 

• The IBM Deskstar™ 16GP [3]. 16.8 GB, 5400 RPM, 162 Mbits/sec Maximum 
internal data rate, 9.5 ms average seek time. 

• Promise Technologies Ultra 33™ PCI EIDE controller card [4]. Supports 4 
drives, Ultra ATA/EIDE/Fast ATA-2. Cost: $50. 

Both the Quantum Bigfoot™ TX 12 GB and the IBM Deskstar™ 16GP 16 GB 
disks were successfully tested with the following systems: 
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System 


Notes 


Mac OS 8.1 on a Macintosh G3 [5] rev. 2 
motherboard 


With HFS+ and both Master/Slave. 


Windows 95b on a Dell Dimension XPS 350 
computer [6] with PhoenixBIOS 


Ok, depending on the BIOS 1 . 
Use FAT 32. 


Windows 98 on a Dell Dimension XPS 350 
computer [6] with PhoenixBIOS 


Ok, depending on the BIOS 1 . 
Use FAT 32. 


RedHat LINUX 5.1 (kernel 2.0.34) on a Dell 
Dimension XPS 350 computer [6] with 
PhoenixBIOS 


Ok, depending on the BIOS 1 . 


Promise Technologies Ultra 33 on a Dell 
Dimension XPS 350 computer [6] with 
PhoenixBIOS 


Ok with Windows 95b and Windows 
98. However, a patch 2 was needed for 
Red Hat LINUX. 



Ten Terabyte EIDE Disk Architecture 

The recipe for a simple 10 Terabyte EIDE Disk Architecture is as follows: 

• Attach eight 16GB EIDE disks to each of 75 CPUs with the help of Promise 
PCI controller cards. 

• Since EIDE cables have a maximum length of 18", it is easier to run extra DC 
power cables into a computer tower than to run EIDE cables out. 

• Load data on these disk arrays. 

• Plan to usually run analysis jobs on the same machine as the data. 

• Use fast Ethernet switches to allow for remote jobs at a modest level. 

Future 

Future plans may include testing the drives with Apple Rhapsody, Sun Solaris, and 
newer releases of Red Hat LINUX. (The 8 GB limit seen so far on Rhapsody DR2 
and Solaris 2.6 may be fixed in later releases.) Also new technologies that are worth 
investigating include both "Lazy RAID" and Firewire™. 

iScc "Getting beyond the ATA 8.4 GB limit" 
http://www.storage.ibm.eom/hardsoft/diskdrdl/library/8.4gb.htm 
http://www.storage.ibm.com/techsup/hddtech/welcome.htm 
http:/ /www. storage. ibm.com/hardsoft/diskdrdl/prod/deskstar. htm 
http: / / www.storage.ibm.com/hardsoft / diskdrdl/prod /ultrastar.htm 
and "8.4 GB Barrier" 

http:/ /www. quantum, com/src/whitepapers/8. 4barrier.html 
http: / / www.quantum.com / products /hdd /bigfoot_tx / 
and "IDE Hard Drive Capacity Barriers" 

http:/ /www. maxtor.com/technology/whitepapers/capbarO. html 

2 Patch available from http://pobox.com~brion/linux/promise34.gz, 
but support is included in kernel 2.0.35 
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Lazy RAID 

Lazy RAID (Redundant Array of Inexpensive Disks) is an idea for using disk arrays 
that offers protection for disks in the event of catastrophic failure of one disk in the 
array This system uses a number of data disks (say 7) plus one parity disk. Therefore, 
if one disk dies the parity disk would allow the recovery of data from the dead disk. 
One could use the RAW DEVICE interface to calculate parity with the CPU. If a disk 
fails then the operator would swap out the dead EIDE drive and reconstitute the dead 
disk drive onto the replacement drive using the parity disk and the remaining data 
disks. This system is well suited for use as scratch disks where a filtered DST is placed 
on disk once and read and analyzed many times. Using this scheme the one parity disk 
is updated only when a file is written to (or erased from) a disk. 

Firewire 

Firewire IEEE 1394 Specifications [7]: 

• Up to 25 or 50 Megabyte/s. 

• Up to 63 devices per interface. 

• Uses two twisted pair data lines. 

• "Fairness" bus arbitration. 

• Supported by MacOS and Windows 98. 

A printed circuit board and DSP driver software would have to be developed using 
the TI chip set . Shown below is a Firewire to EIDE Disk Block Diagram that might 
allow one Terabyte Per PCI Slot: 



Adaptec 8920 
Firewire/PCI Card 



$30 Texas Instruments Chip Set 
39C1011 Firewire/EIDE Bridge 
TMS320F206 DSP Chip 
TSB41LV06 Serial Bus Device 



$30 Texas Instruments Chip Set 
39C1011 Firewire/EIDE Bridge 
TMS320F206 DSP Chip 
TSB41LV06 Serial Bus Device 



$30 Texas Instruments Chip Set 
39C1011 Firewire/EIDE Bridge 
TMS320F206 DSP Chip 
TSB41LV06 Serial Bus Device 




Up to 63 Devices 
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Conclusion 



EIDE disk arrays are an inexpensive way to add large amounts of disk space to both 
single Workstations (and PCs) and multiprocessor computing farms. They provide an 
additional layer to the data storage "cake". 



Speed 
(Megabyte/s) 

1500 
500 



21 



15 



Data Storage Cake 




Media Cost 
($/Gigabyte) 

$15000 
$1300 



$68 
$24 



$2 



[1] S. Bracker, K. Goundcr, K. Hcndrix, and D. Summers, A Simple Multiprocessor Management 
System for Event- Parallel Computing, IEEE NS-43 (1996) 2457. 

[2] http://www.quantum.com/products/hdd/bigfoot_tx/ 

[3] http://www.storage.ibm.com/hardsoft/diskdrdl/desk/1614data.htm 

[4] http://www.promise.com/html/sales/Ultra33.html 

[5] http: / /www. apple.com/powermac/g3 / 

[6] http: / /www. dell. com / products /dim/xpsr /specs /index. htm 

[7] http: / / www. skipst one . com/info . html 

[8] http://www.ti.com/sc/docs/news/1998/98029.htm; 
http:/ /www. ti.com/sc/docs/dsps/details/43/flash. htm; 
http://www.ti.com/sc/docs/msp/1394/411v0x.htm; 
http://www.ti.com/sc/docs/storage/products/cont.htm 
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EIDE/ATA Disk Drive Update - - December 1999 



Costs of disk drives with EIDE/ATA interfaces [1] have fallen since the Chicago 
CHEP '98 conference. Drive capacity and I/O rates have risen. EIDE/ATA disks 
beyond the old 8 Gigabyte limit now work with more platforms and cards. EIDE disks 
remain more than twice as cost effective as SCSI disks and now equal the internal I/O 
speeds of many SCSI disks. We have run RAID 5 on EIDE disks under Linux which 
both stripes data across disks for speed and provides parity bits for data recovery. Tape 
backup may no longer be required to recover from one disk failure in a set. Arrays of 
EIDE disks with Linux PCs serving as disk controllers are attractive and may provide 
many Terabytes of economical rotating online storage. The cost of a quarter Petabyte 
EIDE disk farm is approaching the cost of a StorageTek PowderHorn silo [2] with 
5000 50 Gigabyte RedWood tapes. Finally, we include more information on IEEE- 
1394 Fire Wire which may allow Terabyte arrays of EIDE/ATA disks to be directly 
connected to one or more computers at up to 50 Megabytes/second per interface. 



Table 1. 


Disk 


Maximum 






Unit 






Capacity 


Internal 


Avg. 




Street 


Dollars 


EIDE/ATA 


in 


I/O 


Seek 




Cost 


per 


Disk Drive Model 


Gigabytes 


Mbits/s 


ms 


RPM 


[3] 


Terabyte 


Maxtor DiamondMax 40 [4] 


40.9 


295 


9.0 


5400 


$243 


$6000 


IBM Deskstar 37GP [5] 


37.5 


248 


9.0 


5400 


$302 


$8100 


IBM Deskstar 34GXP [5] 


34.2 


284 


9.0 


7200 


$311 


$9100 


Seagate Barracuda ATA [6] 


28.5 


323 


8.6 


7200 


$211 


$7400 


Western Digital Expert [7] 


27.2 


284 


9.0 


7200 


$236 


$8700 


Western Digital Caviar [8] 


30.7 


271 


9.5 


5400 


* 


* 


Quantum Fireball let 10 [9] 


30.0 


297 


8.9 


5400 


* 


* 


Maxtor DiamondMax Plus [10] 


40.9 


345 


9.0 


7200 


* 


* 



Table 2. 


Disk 


Maximum 






Unit 






Capacity 


Internal 


Avg. 




Street 


Dollars 


SCSI 


in 


I/O 


Seek 




Cost 


per 


Disk Drive Model 


Gigabytes 


Mbits/s 


ms 


RPM 


[2] 


Terabyte 


Seagate Barracuda [11] 


50.1 


264 


7.4 


7200 


$806 


$16100 


Quantum Atlas 10K [12] 


36.4 


315 


5.0 


10000 


$945 


$26 000 


IBM Ultrastar 36XP [13] 


36.4 


231 


7.5 


7200 


$756 


$20 800 


IBM Ultrastar 72ZX [14] 


73.4 


473 


5.3 


10000 


* 


* 


Seagate Cheetah 73 [15] 


73.4 


427 


5.6 


10000 


* 


* 



Announced, but not yet shipping. 
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We now have our 12 GB Quantum Bigfoot TX [16] and 16.8 GB IBM Deskstar 
16GP [17] EIDE drives running in our Sun Ultra 10 workstation. We put in a newer 
motherboard (Sun Part No. 375-0009-09, Date Code: 9843 DARWIN M/B) and up- 
graded the operating system from Solaris 2.6 to Solaris 7. We did not test the two 
changes individually, but one or both put us over the old 8 Gigabyte limit. 

Promise Technology now sells their Ultra66 [18] EIDE to PCI controller card. The 
Ultra66 provides up to 66 MB /s on each of two channels with two drives per channel. 
The card costs $40. We use Promise's previous 33 MB/s EIDE to PCI card daily in 
our Linux PC server (mail, backup...) with IBM Deskstar disks [19]. 

ProMAX [20] sells a TurboMAX/ATA 33 Host Adapter for Macintosh PCI buses. 
It allows adding four EIDE drives to an Apple Macintosh. Disks can be striped in 
pairs. FirmTek [21] is working on an Apple Macintosh software driver for the Promise 
Ultra66 EIDE to PCI card. 

Terabytes of Linux RAID 5 Disks 

RAID [22] stands for Redundant Array of Inexpensive Disks. Many industry of- 
fering meet all of the qualifications except the inexpensive part, severely limiting the 
size of an array for a given budget. This may change. RAID on EIDE disks under 
Linux software which both stripes data across disks for speed and provides parity bits 
for data recovery (RAID 5) is now available [23]. With redundant disk arrays, tape 
backup is not needed to recover from the failure of one disk in a set. This removes a 
major obstacle to building large arrays of EIDE disks. A RAID 5 set of eight 41 Giga- 
byte disks fits in a full tower case of a PC running Linux. This provides over a quarter 
of a Terabyte per box. The boxes would be connected using 100 Megabit /second Fast 
Ethernet PCI cards in each box plus Ethernet switches [24, 25]. This looks to be very 
doable. 

We have done a quick test of the Linux RAID 5 software using two 25 Gigabyte 
IBM Deskstar 25GP [19] EIDE disks. The host was a Pentium II with Red Hat Linux 
6.0 [26] and a Promise Technology Ultra 33 EIDE/PCI card. The test ran as expected. 
Naturally, half the disk space is devoted to parity with only two disks. For a real RAID 
5 system, eight or more disks would be a more efficient use of space. The fraction of 
disk space devoted to parity equals the inverse of the number of disks in a set. 

On a mundane note, disk drives typically draw two amps at 12 volts for 15 seconds 
when starting. Thus eight drives can draw 16 amps at 12 volts. This can tax the 
ratings of an inexpensive commodity 300 watt PC power supply. Care needs to be 
taken to choose a supply with a large portion of its wattage devoted to 12 volts. The 
U.S. EPA Energy Star/Green PC Initiative has led to the development of a Standby 
command for disks that might allow a staggered startup of a disk array. The command 
"/sbin/hdparm -S n" will spin down disks under Linux. As array size grows, a second 
commodity 300 watt PC power supply might be required per case. 
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Table 3: Power consumption of large EIDE disk drives. 













Read/ 








Startup 




Write/ 




EIDE/ATA 




5V 


12V 


Seek 


Idle 


Standby 


Disk Drive Model 


GB 


Amps 


Amps 


Watts 


Watts 


Watts 


Maxtor DiamondMax 40 [4] 


40.9 


0.3 


2.1 


11.0 


5.2 


1.3 


IBM Deskstar 37GP [5] 


37.5 


1.0 


2.0 




4.9 




IBM Deskstar 34GXP [5] 


34.2 


1.0 


2.0 




6.9 




Seagate Barracuda ATA [6] 


28.5 


0.7 


2.4 


9.7 


7.0 


0.8 


Western Digital Expert [7] 


27.2 


0.7 


2.0 


10.1 


6.9 


0.8 


Western Digital Caviar [8] 


30.7 


0.6 


1.8 


10.0 


6.2 


1.3 


Quantum Fireball let 10 [9] 


30.0 


0.7 


1.7 




5.0 




Maxtor DiamondMax Plus [10] 


40.9 


0.6 


2.5 


12.9 


6.7 


1.4 



Comparison of Quarter Petabyte Disk and Tape Storage Systems 

In Table 4, we compare a quarter Petabyte EIDE disk farm to an automated Stor- 
ageTek PowderHorn tape silo [2] with eight RedWood tape drives and 5000 50 Gigabyte 
tapes. The disk farm estimate includes disks, parity disks, Linux and RAID software, 
CPUs, motherboards, cases [27], power supplies, memory, Fast Ethernet PCI cards, 
Promise Ultra66 cards, Ethernet switches [24, 25], and racks. The Linux PC that runs 
each disk set costs about the same as a high end SCSI-to-PCI controller card. 

To achieve a quarter Petabyte, 873 Linux PCs are required with eight 40.9 GB 
disks each. One eighth of the disk space is devoted to parity for data recovery from 
disk failure. Care must be taken to write protect files and disks to prevent accidental 
deletion. Physically, the PCs form a wall 4 high by 2 deep by 110 wide (2.4 x 1.1 
x 24 meters). Each Linux PC consumes about 90 watts, equally divided between the 
disks and the CPU/motherboard. A dozen 24 000 BTU window air conditioners would 
suffice to remove this 80 Kilowatt heat load. Much less heat is generated in standby 
mode. The first level network consists of 288 $75 fast ethernet switches [25]. A single 
high end switch with 288 fast ethernet ports is used for the network backbone [24] . The 
disks themselves can be used to transport data between sites. A high rate experiment 
might generate a Terabyte of data a day which one wished to move. A Terabyte fits 
on 25 disks, which easily fit in a suitcase for shipping. 

In summary, the disk farm cost is not too much greater than the tape silo cost and 
the performance of the disks is far better. One also gets a Teraflop of computing power 
[28] as a free bonus; and the disk farm encapsulates data with instructions in physical 
computing objects which can be exploited to increase efficiency Disk farm sizes can 
be scaled in size with great flexibility! Its sometimes difficult for a university to buy a 
whole tape silo [29] . Now everyone can have the benefit of online data. 
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Table 4: Comparison of a 250 TB EIDE disk array to a 250 TB StorageTek tape silo. 











No. of 


I/O 


Time 








Avg. 


Avg. 


Disk or 


per 


to 




Storage 


Vol. 


Power 


Access 


Tape 


Drive 


Read 


Purchase 


System 


m 


Kw 


Time 


Drives 


MB/s 


250 TB 


Price 


EIDE RAID Disks 


100 


80 


0.009 s 


7000 


20 


34 minutes 


$2100 000 


STK Tape Silo 


25 


6 


115 s 


8 


11 


32 days 


$1300 000 



Terabyte Arrays of IEEE-1394 Fire Wire Disks 

The amount of disk space one can connect to a CPU directly with 18" EIDE cables 
is currently less than a Terabyte. In some applications, one might want to access more 
data with fast local disk and not suffer the overhead of network software. One may 
be able to use four IEEE-1394 FireWire buses [30] with a single CPU to attach up 
to 63 inexpensive EIDE disks per bus for a total of 10 Terabytes of local storage at 
200 MB/s. FireWire's peer-to-peer topology also adds significant new functionality 
by allowing multiple computers to share multiple disks directly on a single bus. 

Symbios Logic/LSI Logic has a $13 IEEE-1394 to ATA/ATAPI controller chip 
[31]. The SYM13FW500 integrates a 400 Mbits/s IEEE-1394 (FireWire) physi- 
cal interface (PHY) with an ATA/ATAPI interface, all on a low-power CMOS IC. 
Each SYM13FW500 supports two ATA/ATAPI devices. Wyle Electronics distributes 
the part. Recently, Oxford Semiconductor has also introduced an IEEE-1394 to 
ATA/ATAPI controller chip, the OXFW900 [32]. Texas Instruments has decided not 
to market their prototype IEEE-1394 to ATA/ATAPI chip. 

EIDE disks with EIDE to IEEE 1394 FireWire interfaces are available from LaCie 
[33] and VST Technologies [34]. VST Technologies has also shown FireWire RAID 
arrays with mirroring and striping, but not parity [35]. All the disks are EIDE. 

An interesting possibility might be to put eight EIDE drives in an inexpensive PC 
case with a 300 watt power supply. Then add four EIDE to FireWire interface chips 
to a circuit board with the same form factor as a PC motherboard. The color of the 
PC case can even be special ordered [27] for Apple Macintosh users. 

Andreas Bombe, Sebastien Rougeaux, and Emanuel Pirker are in the process of 
writing GNU Linux software drivers for IEEE 1394/FireWire devices [36]. 

Pieces appear to be converging. It may, in the not too distant future, be possible 
to directly connect Terabyte arrays of EIDE/ATA disks to one or more computers 
at up to 50 Megabytes/second per FireWire interface. Some computers come with 
FireWire on the motherboard. FireWire interfaces can also be added with cards such 
as OrangeMicro's HotLink FireWire PCI Board [37]. 
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http://www.hawkingtech.com/ PN505ES $79 

http://www.linksys. com/scripts/features. asp?part=EZXS55W EtherFast $90 
http://netgear.baynetworks.com/products/fsl05ds.shtml FS105 $83 

http://smc.com/smc/pages_html/switch.html EZNET-5SW/6305TX $78 



[26] http://www.redhat.com 

[27] IW-Q600 ATX Full Tower Case, 11 bays, 300 watt power supply - $77 
Q600 case dimensions: 600mm high x 200mm wide x 432mm deep. 
IW-Q2000 ATX Full Tower Case, 11 bays, two 300 watt power supplies - $208 
Q2000 case dimensions: 600mm high x 200mm wide x 476mm deep. 
'Available in different color for OEM customers." 
http: / /www. in-win.com/framecode/index. html 
http: / / www.pricewatch.com 

[28] Advanced Micro Devices and Pentium III CPUs can perform four single precision 
floating point adds or multiplies per clock cycle with their 3DNow! or Streaming 
SIMD Extensions units, respectively. Both 3DNow! and SSE are implementations 
of SIMD (Single Instruction, Multiple Data) processors. 
http://www.amd.com/products/cpg/athlon/index.html 
http: / /www. amd.com/products/cpg/k6iii/index. html 
http: / /www. amd.com/products/cpg/k623d/index.html 
http: / /www. intel.com/home/prodserv/pentiumiii/prodinfo. htm 
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[29] For situations with more people than money, manually loaded tapes provide the 
way to store and move data with the lowest initial investment. The lowest media 
cost is given by 112 meter long 8mm tapes storing 5 Gigabytes uncompressed on 
an Exabyte Eliant 820 at 1 MB/s. The tapes cost 53 cents per Gigabyte at the 
Fermilab stockroom and the Eliant 820 tape drive costs $1300. Used Exabyte 
8500 and 8505 tape drives are even cheaper on ebay.com. Using pairs of drives, 
with one running and the other waiting on deck with a tape ready to go, gives 
operators time to load tapes [38, 39]. 
http: / / www.exabyte.com/products/8mm/eliant / 
http: / /www-stock. fnal.gov/stock/ 
http://www.exabyte.com/products/ 

[30] IEEE Standard for a High Performance Serial Bus, ISBN 1-55937-583-3. 
http://standards.ieee.Org/catalog/bus.html#1394-1995 
http: / / standards.ieee.org/reading/ ieee / std_public/ description 
/busarch/ 1394- 1995_desc.html 

[31] SYM13FW500 ATA/ATAPI to 1394 Native Bridge Data Manual Version 1.02 
ft p : / /ftp . symbios. com / pub / symchips / 1 394 
http:/ /www. symbios.com/news/pr/80330ata.htm 

[32] http: / / www.oxsemi.com/ pro ducts/pro ducts, html 

[33] http: / /www.lacie.com/scripts/harddrive/ drive. cfm?which=30 

[34] VST Technologies, 125 Nagog Park, Acton, MA 01720 978-635-8282 
http: / / www.vsttech.com/vst / products. nsf / pLfirewire 
http: / /www. macnn.com/thereview/reviews/vst/fwhd.shtml 
http://www.elgato.com/products.html FireWire Disk Control 1.01 

[35] http://www.vsttech.com/vst/press.nsf/default 08/31/99 09/17/99 
http: / / www.softraid.com 

[36] http:/ /eclipt.uni-klu.ac.at/ieeel394/ 

http://www.kt.opensrc.Org/ktl9990722_28.html#15 

[37] http: / / www.orangemicro.com/ firewire.html 

[38] FNAL E769: C. Stoughton and D. Summers, Using Multiple RISC CPUs in 
Parallel to Study Charm Quarks, Comput. Phys. 6 (1992) 371; C. Gay and S. 
Bracker, IEEE NS-34 (1987) 870; S. Hansen et al, IEEE NS-34 (1987) 1003; 
G. Alves et al., Phys. Rev. Lett. 69 (1992) 3147; 77 (1996) 2388; 77 (1996) 2392; 
Phys. Rev. D56 (1997) 6003. 

[39] FNAL E791: S. Bracker, K. Gounder, K. Hendrix, and D. Summers, A Simple 
Multiprocessor Management System for Event- Parallel Computing, IEEE NS-43 
(1996) 2457; S. Amato et al., Nucl. Instrum. Meth. A324 (1993) 535; 
E. M. Aitala et al., Phys. Rev. Lett. 76 (1996) 364; 81 (1998) 44; hep-ex/9809026; 
hep-ex/9809029; hep-ex 9912003. 
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